Evaluating generative patent language models

Generative language models are promising for assisting human writing in various domains. This manuscript aims to build generative the patent domain and evaluate model performance from a human-centric perspective. The perspective is measure ratio of keystrokes that can be saved by autocompletion based on models. A higher means more effective which save keystrokes. metric used benchmark performance. different conventional machine-centric metrics token-based instead keystroke-based. In terms size, largest built this 6B, state-of-the-art domain. Based metric, it found not necessarily best metric. finding keeping increasing sizes might unnecessary if purpose assist with autocompletion. Several pre-trained scratch research. released future researchers. visualization tools also provided. importance building potential facilitate creativity innovations future.